A video is a sequence of images (also referred to as “frames”), that may be encoded and compressed, such as, for example, as motion-JPEG (M-JPEG) or according to the H.264 standard. The terms ‘frame’ and ‘image’ are used interchangeably throughout this specification to describe a single image in an image sequence, wherein the image sequence includes one or more images. An image is made up of visual elements, for example pixels or 8×8 DCT (Discrete Cosine Transform) blocks, as used in JPEG images in a motion-JPEG stream.
A global luminance change is a change that affects the visual representation of a large part of a scene captured by an image sensor. The change may arise, for example, from a change in the strength of a lighting source illuminating the scene or a change in one or more settings of the sensor. A global luminance change is often not associated with a change of the semantics of the scene. That is, the visual representation of the scene changes even though the actual scene has not changed. In practice, a luminance change is often limited only to a portion of the image when, because of multiple light sources and/or obstacles, not all parts of the scene are affected equally by the luminance change.
Global luminance changes may be caused by a change in the scene, by a change in the sensor parameters, or by contemporaneous changes in both the scene and the sensor parameters. A change in the scene may occur as a result of a change in lighting, for example. A change in the sensor parameters may occur as a result of changes to camera gain, aperture, focus adjustment, or any combination thereof.
Luminance compensation can be used on image sequences to reduce the effects of global luminance changes on the image sequence. Luminance compensation involves using low frequency luminance components of an image in an image sequence for a particular purpose, for example, to reduce flicker in a sequence of images intended for human visualisation. Thus, luminance compensation can improve the stability of the DC (zero frequency) luminance components of an image, where the image content is unchanged and the DC luminance components of the unchanging image content are modified only due to a change in luminance across the whole scene.
Global luminance changes pose a problem for background modelling and background-differencing techniques, since regions of the image that would have been similar to the background become different from the background model due to the luminance change. The background modelling systems, using background-differencing, are dependent upon the low frequency (close to DC) luminance components of an image. Accordingly, changes to the luminance components across the entire scene, due to changes in lighting or camera settings, can cause false detections of foreground in a scene. Thus, since the differences between block and mode model are high according to their visual representations, regions that are really background (the same as before) are now classified as foreground, due to the difference in their luminance values. For this reason, a background modelling system is more sensitive to long term changes in luminance values across an entire scene than the human vision and the video compression systems. For example, video compression systems are typically more concerned, from the point of view of video coding efficiency, with what looks similar than what is actually similar. That is, if a luminance change is hardly perceptible to a viewer, systems that are concerned with image communication and display have no interest in correcting for the change. For background modelling, however, a small change can translate to big differences during further processing. For example, it can be the difference between classifying foreground/background.
Background modelling systems may incorporate techniques to update a scene model based on the latest observed image. The update techniques allow background modelling systems to cope with structural changes in the scene. For example, a new painting in a museum will initially be detected as a foreground object, but after some time the painting becomes part of the background. The same update techniques allow background modelling systems to deal with lighting changes. Temporarily, the regions affected by luminance changes are considered foreground, but after a while the regions become background again.
One problem with the use of such model updating techniques is that it takes time to adjust to the new situation. For fast changes in luminance, the resulting false detections are undesirable. Another problem is that only the models of modes that are matched are being updated. However, luminance changes can occur when part of the background (represented by a first mode model) is occluded by foreground (represented by a second mode model). When the foreground object moves away and the background region is revealed again, the region should be classified as being similar to the background model, even though the global luminance changes have made the revealed region of the image look different from the background model.
Other luminance compensation approaches calculate the mean luminance value of each image in an image sequence and track the mean luminance value of each image in an image sequence using an exponential moving average. Some approaches prefer tracking the median luminance value, as the median value is less dependent on local changes in the frame.
The approaches which track the mean (or median) value of the image assume that changes in the mean (or median) value of frame are due to global lighting changes. This assumption is not valid in scenarios where a foreground object occludes part of the scene and the foreground object has different luminance characteristics from the occluded region. In such scenarios, these approaches are not sufficiently robust.
Some luminance compensation methods operate on individual images in the image sequence. Such methods may, for example, use histogram-based measures to compensate for luminance changes in an individual image. These methods are very sensitive to noise caused by non-representative frames and temporary changes in the scene that are not caused by lighting.
Some luminance compensation approaches modify the luminance compensation when some measure of the overall luminance of the scene moves beyond a certain range. The range may be either predetermined, or dependent on the previous luminance values in the image sequence.
Thus, for background modelling applications, an accurate luminance compensation method is desired that minimises false foreground detections caused by lighting changes. In addition, for real-time background modelling and background-differencing applications, it is preferable for the utilised luminance compensation method to have high performance and predictable results. For use in embedded systems, or on servers serving a large camera network, a luminance compensation method should have relatively low computational requirements, so that it does not interfere computationally with the actual background modelling and background-differencing process.
Thus, there is a need to provide an improved luminance compensation method and system.